Flight training and synthetic visualization system and method

ABSTRACT

A low-cost training and synthetic visualization system and method directed to improving an individual&#39;s airborne performance in general aviation, skydiving, and other aerial applications. The system is comprised of a self-contained mobile sensor and data storage device for recording the travel path, orientation, and forces acting upon an object as it moves through space, a desktop graphics software program for creating a playback of the recorded data on a three-dimensional representation of the environment through which the object moved, a means of linking the sensor and data storage device to the software program for the purpose of exchanging information, and a centralized data storage and retrieval system designed to accept, assimilate and redistribute the recorded data.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a continuation of and claims the benefit of U.S.patent application No. 11/327,965, entitled “Flight Training andSynthetic Visualization System and Method,” filed Jan. 9, 2006, now U.S.Pat. No. 7,848,698, issued Dec. 7, 2010, and U.S. Provisional PatentApplication No. 60/701,736, entitled, “Low-Cost Flight Training andSynthetic Visualization System,” filed Jul. 22, 2005.

FIELD OF INVENTION

This invention pertains to a low-cost system and method for providingflight training through the use of a self-contained mobile dataacquisition, recording and storage unit that takes quantitativemeasurements of an airborne object's movement and orientation in athree-dimensional space, and the subsequent processing and playback ofsaid measurements.

BACKGROUND

Various methodologies have been developed that provide flight trainingand/or analysis of pre-recorded activities. One methodology provides arealistic, three-dimensional software simulation of flight in order toallow pilots to practice flight techniques without actually flying in anairplane. An example of this methodology is the software program called“Flight Simulator” by Microsoft Corporation. In this and other similarflight simulation programs, a user can complete a simulated flight andthen play the simulation back to analyze their performance. Programs ofthis nature provide realistic simulations of flight in an artificiallygenerated three-dimensional environment in which aircraft behaviors aremodeled quite accurately with respect to the physics of flight. Howeverreal the simulation may appear, the information produced is still only asimulation and can not provoke the behaviors and responses of a studentin a real airplane in a real life training situation whose behavior haslife and death consequences. Neither can a simulation provide thesensory perception imparted to a person in flight by an actual airplanethat is acted upon by external stimulations such as weather, loading,and altitude.

Inventors have developed full-motion or partial-motion flight simulatorsystems that attempt to improve on software-only flight simulators. U.S.Pat. No. 6,634,885 B2, issued to Hodgetts et al., describes a systemthat mounts a simulated aircraft flight deck onto a motion platform thatis moved by electric motors to recreate the motions one would feel in anactual aircraft. This system can be coupled with and controlled by aflight simulator program such as Microsoft Flight Simulator.

U.S. Pat. No. 4,527,980, issued to Miller, describes a flight simulatingvideo game system that uses an aircraft-shaped enclosure resting on aparabolic dish to produce pitch and roll movements based on theoperator's movements of the flight controls. A monitor inside theenclosure displays simulated flight images that are oriented based onthe current position of the aircraft-shaped enclosure to simulate theview through an aircraft window.

The addition of movement and tactile feedback is a distinct improvementover a software-only system for flight training, but demands a complex,bulky, and expensive electro-mechanical platform to add even thesimplest motion, making it impractical for private home use.

Another category of inventions includes inertial measurement units(IMUs) which are permanently mounted in an aircraft and which takemeasurements on the aircraft's movements through space. The mosteffective of these devices are those which combine sensors (such asaccelerometers and gyroscopes) that measure inertial movement withglobal positioning system (GPS) sensors to measure latitude, longitude,and altitude. Although these devices are not designed to be flighttraining systems, the data they produce can be useful in flight trainingsituations.

U.S. Pat. No. 6,480,152 B2, issued to Lin et al., and its relatedapplications describe a micro-system which integrates a separate IMUwith a GPS chipset and magnetic field sensor to produce highly-accuratedata relating to flight which can be off loaded to an external system.This device will generate information about the aircraft includingposition (in terms of latitude, longitude, and altitude), orientation(in terms of yaw, pitch, and roll), and magnetic heading. One of thedrawbacks of this invention is that it does not have its ownrechargeable power source, and must be direct-wired into a vehicle'spower supply. It is not a self-contained, mobile device with an integralset of user controls and feedback devices. This prevents the device frombeing quickly moved from vehicle to vehicle or from vehicle to home, anddoes not allow for use on a human body. The invention claimed does notstore the data it records for later transmission to and processing by aseparate analysis system, but sends it immediately to a user interface.The claimed invention does not include a separate component for theprocessing and display of the information that is captured by thedevice. Although the invention has usefulness as an aircraft instrumentand data source, its usefulness as a flight training system is limited.

Atair Aerospace of Brooklyn, NY, provides a portable data acquisitionunit which combines GPS and an IMU to record navigation information.This stored information can be later downloaded using a direct wiredconnection to another system. A separate desktop software applicationallows the user to display the recorded data and view simpletwo-dimensional and three-dimensional graphs of the data. This systemdoes not provide integrated user controls, but is instead activated by aremote switch. This system does not have an integrated power source andcharging circuit, and requires an external battery pack or power source.The data acquisition unit cannot be quickly moved from one applicationto the next, and is not designed to be used on a human body.

Eagle Tree Systems, LLC, of Bellevue, WA, offers a data recording systemfor radio controlled (RC) aircraft that can track and transmit severalperformance parameters for the aircraft, including speed, engine RPM,and the positions of the servo motors controlling the various flightsurfaces. This data can be transmitted to the operator of the RCaircraft, who can use the data to monitor the flight. Additional datacan be added by plugging in a separate GPS module which can provideposition data for the aircraft. This GPS position data can be used toprovide a crude playback of the completed flight. The GPS module is notan integral part of the main flight recorder and must be purchasedseparately. The system does not provide information on the orientationof the aircraft (that is, the current yaw, pitch, and roll of thevehicle), and does not have an inertial measurement unit or alternatemeans of position detection when the GPS signal is lost. The mainfunction of the system is to track engine and aircraft performanceincluding the position of the servo motors. The Eagle Tree system isintended for use on unmanned vehicles only and is not a manned flighttraining system.

A third category of inventions includes systems which are designed tomeasure the movement of a body through three-dimensional space and tocreate a playback of that movement on a separate external system. Thereferenced patents are not flight training systems, but describe systemsthat can be used to facilitate training in other applications throughthe measurement of a moving object.

U.S. Pat. No. 6,885,971 B2, issued to Vock et al., describes severalmethods and systems for measuring the various performance parametersassociated with extreme sports. Data on parameters is collected by a setof sensors that can include a microphone system for detecting vibrationand shifts in unit speed, an accelerometer for detecting changes inmovement, and pressure sensors for detecting changes in altitude. Thedata is collected by a sensor or group of sensors located on the bodyduring an event, and transmitted to a base station where the Internet isused to view the data. This invention is designed to measure performanceparameters such as “air time” (the length of time a body remains off theground), “drop distance” (the vertical distance covered by an athletegoing over a jump or drop-off), and “power” (the total number ofg-forces experienced by the athlete during a performance). Thesemeasurements are gathered by sensors which require interaction with theground (measuring vibration, sound, and sudden acceleration changes) andare not suited for use on an aircraft. The invention does not have amethod for determining position (latitude and longitude), and has nomethod for measuring the orientation (yaw, pitch, and roll) of a movingbody.

WIPO Pat. No. WO 2005/053524 A1, issued to Limma et al., describes amethod and system for measuring information from an activity anddisplaying feedback on that activity to at least one individual. Thissystem relies on the signal from a GPS receiver to determine anindividual's position (latitude and longitude) and altitude. In additionto the GPS position, the sensor for this system may include a barometerand thermometer for measuring ambient pressure and temperature, and aheart rate monitor for measuring the heart rate of the individual duringthe activity. This system is not designed to be mounted in an aircraftor other airborne vehicle. There is no means of inertial measurement,and therefore no direct means to determine the orientation (yaw, pitch,and roll) of the moving body.

WIPO Pat. No. WO 2005/053528 A1, also issued to Limma et al., is basedon an invention similar to that described in WO 2005/053524 A1, butfurther provides a method for comparing the performance in a previousevent to the ongoing performance in the current event. The systemdisplays feedback in the form of an ongoing comparison of the twoevents, and allows a performer to see if they are matching or exceedingthe previous performance. As with the previous patent described (WO2005/053524 A1), this invention is not designed to be used in anaircraft or other airborne vehicle, and provides no means of inertialmeasurement.

U.S. Pat. No. 5,173,856, issued to Purnell et al., describes a vehicledata recording system used for recording measurements from on-vehiclesensors. The primary application of this system is in automobiles andautomobile racing. This system is capable of logging measurements inmemory and later displaying these measurements against a second set ofmeasurements so that the two sets can be compared to highlightdifferences. This system is not fully self-contained, and relies onobtaining data from existing on-vehicle sensors, as well as sensorspermanently mounted on the vehicle course or racetrack. The system doesnot provide the three-dimensional position or orientation of thevehicle, but merely records data from the aforementioned sensors. Thesystem is designed to be permanently mounted in a vehicle, and tied tothat vehicle's systems, and cannot be quickly moved to another vehicleor attached to a human body.

Many of the inventions described herein rely on the permanent mountingand integration of the electronic sensors into a vehicle system, whichprevents the sensors from being quickly ported to other variedapplications. Other inventions are mobile and can be used to recorddata, but are based on limited sensing capabilities that do not fullycapture the movements or position of a moving body. The known solutionsreferenced herein do not describe a flight training and syntheticvisualization system or method which comprises a fully mobile andself-contained data recording unit, a software means for creating aplayback of the recorded trip, a means of linking the mobile datarecording unit to the software means for the purpose of exchanginginformation, and a centralized database designed to accept recorded tripdata.

SUMMARY OF THE INVENTION

Accordingly, it is a main objective of the present invention to describea flight training and synthetic visualization system which comprises afully mobile, self-contained data recording unit, a desktop graphicssoftware engine for creating a playback of the recorded trip, a means oflinking the mobile data recoding unit to the software engine for thepurpose of exchanging information, and a centralized data storage andretrieval system designed to accept and assimilate recorded trip dataand distribute pertinent data to system users.

It is another objective of the present invention to describe a method offlight instruction and analysis in which navigational data is capturedby a mobile data recording unit and stored in the mobile data recordingunit's memory to be transmitted an indefinite amount of time later forprocessing and display on an external computer system.

It is another objective of the present invention to describe a means ofprocessing and displaying the information received from the mobile datarecording unit by creating a three-dimensional playback of the recordedtrip on a realistic, simulated representation of the actual environmentin which the data was captured.

It is another objective of the present invention to describe a method offlight training in which navigational data is captured by a mobile datarecording unit and transmitted for immediate display in real-time on ahandheld computing device or mobile computer located in close proximityto the mobile data recording unit.

Finally, it is another objective of the present invention to describe amethod of flight training in which navigational data is captured by amobile data recording unit and transmitted for immediate display inreal-time on a computer system at a remote location.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings constitute a part of this specification and includeexemplary embodiments of the present invention illustrating variousobjects and features thereof

FIG. 1 shows a perspective view of a small, self-contained mobilesensor, which is one component of a flight training and syntheticvisualization system described herein.

FIG. 2 shows an embodiment of the flight training and syntheticvisualization system described herein.

FIG. 3 shows an alternative embodiment of the flight training andsynthetic visualization system described herein.

FIG. 4 shows an embodiment of the decal and switch panel for the mobilesensor.

FIG. 5 shows an exploded perspective view of the mobile sensor,highlighting the main components.

FIG. 6 shows a block diagram of the preferred embodiment of theelectronic architecture for the mobile sensor.

FIG. 7 shows an example of a representative graphical user interface(GUI) for a flight analysis application that executes on a separatedesktop or handheld computer.

FIG. 8 shows the same example graphical user interface (GUI) as shown inFIG. 7 with changes to represent how the flight analysis applicationmight appear when the data is displayed in two-dimensional mode, orgraph mode.

FIG. 9 shows the same example graphical user interface (GUI) as shown inFIGS. 7 and 8 with changes to represent additional graphical featuresavailable during the three-dimensional (3D) playback.

FIG. 10 is a high-level flowchart showing the flow of control requiredon the mobile sensor, the desktop application running on the desktopcomputer or the handheld computing device, and the centralized serverduring a typical record and playback cycle.

FIG. 11 provides a definition of the term yaw, and shows a top view of amoving body such as an aircraft.

FIG. 12 provides a definition of the term pitch, and shows a side viewof a moving body such as an aircraft.

FIG. 13 provides a definition of the term roll, and shows a front view.

DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a perspective view of a small, self-contained mobile sensor10, which is one component of a flight training and syntheticvisualization system described herein. The mobile sensor is contained inan enclosure 19, which provides environmental protection for theelectronics which comprise the mobile sensor. A decal and switch panel11 is adhered to the front surface of the enclosure 19, and provides aplurality of user interface switches 12, a plurality of indicator lights13, and a surface 14 for a company logo or other printed matter. Themobile sensor contains a power connector opening 15 which accepts a jackfrom a recharging system. An external antenna 16 extends from the top ofthe mobile sensor for improved reception of satellite signals. Anoptional memory card slot 17 is provided for the use of removable memorydevices such as a memory card 18.

FIG. 2 shows an example embodiment of the flight training and syntheticvisualization system described herein. A mobile sensor 10 is mounted onan aircraft or other moving body and used to collect data about themovement of that body through space. This data may then be transferredby a transfer means 21 in real-time or asynchronously at a later time toa computer 20. The transfer means 21 may comprise a direct-wiredconnection, a wireless connection, or the transfer of data via aremovable memory device. Software on the computer 20 is used to processand replay the data for the operator. The computer 20 can augment theplayback of the data collected by the mobile sensor 10 by downloadingsatellite images and other information from a centralized database 22over an internet-style connection 23. In this embodiment, the primarypurpose of the flight training and synthetic visualization system is theplayback and post-analysis of recorded flight data.

FIG. 3 shows an alternative embodiment of the flight training andsynthetic visualization system described herein. A mobile sensor 10 ismounted on an aircraft or other moving body and used to collect dataabout the movement of that body through space. This data is thentransferred in real-time over a wireless connection 31 to a handheldcomputer or other mobile computing device 30 for immediate viewing bythe operator. In this embodiment, the primary purpose of the flighttraining and synthetic visualization system is to provide real-time,immediate feedback to the operator or instructor on an ongoing flight ortrip.

FIG. 4 shows an example embodiment of the decal and switch panel 11 forthe mobile sensor 10. It is not the intent of this figure to limit thedecal and switch panel functions to those shown, but rather to show onepossible embodiment of the user interface for illustration purposes. Inthis embodiment, the decal and switch panel 11 comprises a Record buttonand indicator light 41 for starting and stopping the data recordfunction, a Lock button and indicator light 42 for locking the keypadagainst inadvertent key presses, a Radio button and indicator light 43for initiating wireless data transfers, a Calibrate button and indicatorlight 44 for calibrating the mobile sensor 10, and an on/off button andindicator light 45 for turning the mobile sensor 10 on and off The decaland switch panel 11 further comprises a Charge indicator light 46 forindicating battery charge, a GPS indicator light 47 for indicatingsatellite connection, and a company logo 40.

FIG. 5 shows an exploded perspective view of the mobile sensor,highlighting the main components. A top enclosure piece 50 provides asurface for the decal and switch panel 11 and serves as the top half ofa protective enclosure surrounding the electronics. An input/output(I/O) circuit board 51 comprises circuitry for detecting operator buttonpresses from user interface switches 12 and houses the indicator lights13. A power supply board 53 comprises circuitry for providing power tothe electronics in the box and regulating any external power source thatis supplied to the mobile sensor during charging. Sandwiched between theI/O board 51 and the power supply board 53 is a rechargeable powersource 52 such as a battery. A satellite receiver board 54 comprisescircuitry for receiving signals from satellite navigation systems suchas the global positioning system (GPS). The satellite receiver board 54also comprises an antenna means 16 to provide for the reception ofsatellite signals. A microprocessor board 56 comprises a microprocessorand related circuitry for overall control of the mobile sensorelectronics. The microprocessor board 56 also comprises circuitry thatallows the mobile sensor to sense rotation about its yaw axis. Attachedto the microprocessor board 56 is the roll board 56A, which allows themobile sensor to sense rotation about its roll axis, the pitch board56C, which allows the mobile sensor to sense rotation about its pitchaxis, and the communications board 56B, which comprises the circuitrynecessary to allow the mobile sensor to communicate with a computer. Theroll board 56A and the pitch board 56C are mounted perpendicular to eachother and to the microprocessor board 56 in order to enable the mobilesensor to sense angular speed and rotation in each of three separateplanes. A bottom enclosure piece 57 serves as the bottom half of theprotective enclosure surrounding the electronics.

FIG. 6 shows a block diagram of the preferred embodiment of theelectronic architecture for the mobile sensor 10. At the highest level,the mobile sensor 10 comprises a microprocessor board 56, a roll board56A, a pitch board 56C, a communications board 56B, a satellite receiverboard 54, an input/output board 51, a rechargeable power source 52, apower supply board 53, and a decal and switch panel 11. These functionalblocks are described in additional detail in the following paragraphs.

The microprocessor board 56 includes a yaw accelerometer 600 for sensingthe magnitude of acceleration of the mobile sensor 10 about its yawaxis, and a yaw gyroscope 601 for sensing the rate of rotation of themobile sensor 10 about its yaw axis.

The signal output by the yaw accelerometer 600 is sensitive to changesin ambient temperature. Temperature and gain compensation are providedby block 603 to correct this signal in various temperature conditionsand to apply a gain multiplier to increase the amount of usefulresolution available from the yaw signal. An analog-to-digital (A/D)converter 602 converts the analog yaw accelerometer 600 signal to adigital signal that can be used by the microprocessor 606. The A/Dconverter 602 also converts the analog yaw gyroscope 601 signal to adigital signal that can be used by the microprocessor 606.

The microprocessor board 56 further includes an XY magnetoresistivecompass 604A for measuring the Earth's magnetic field in both the X andY planes of movement, and a Z magnetoresistive compass 604B formeasuring the magnetic field in the Z plane.

The magnetoresistive compasses 604A and 604B each contain an elementwhich senses its orientation relative to the earth's magnetic field andwhich produces a differential voltage output based on its orientation inthe magnetic field. These differential voltage outputs are sent todifference amplifiers 605, which amplify the outputs to useful voltagelevels. The amplified output voltages are then sent to the A/D converter602, which converts the analog signals from 604A and 604B to digitalsignals that can be used by the microprocessor 606. A pulse resetfeature 604C sends a current pulse to the magnetoresistive compasses604A and 604B periodically to remove any magnetic disturbances which mayhave built up on the sensing elements.

A boundary scan test interface circuit 607 such as JTAG is provided as ameans of programming the microprocessor 606 and as a means of accessingand testing various unit features.

A storage device 609 such as a NAND flash memory module or a removablememory card is used to store the data collected by the microprocessor606 until the data can be downloaded to a separate system. A voltagelevel translator 608B converts the voltage levels output by the storagedevice 609 into levels which can be used by the microprocessor 606, andvice versa. A second voltage level translator 608A is used to convertvoltage levels between the microprocessor 606 and the satellite receiverboard 54 and the wireless radio board 56B.

The roll board 56A includes a roll accelerometer 610 for sensing themagnitude of acceleration of the mobile sensor 10 about its roll axis,and a roll gyroscope 611 for sensing the rate of acceleration of themobile sensor 10 about its roll axis.

Temperature and gain compensation is provided for the roll accelerometer610 by block 613. An analog-to-digital (A/D) converter 612 converts theanalog roll accelerometer 610 signal to a digital signal that can beused by the microprocessor 606. The A/D converter 612 also converts theanalog roll gyroscope 611 signal to a digital signal.

The pitch board 56 includes a pitch accelerometer 620 for sensing themagnitude of acceleration of the mobile sensor 10 about its pitch axis,and a pitch gyroscope 621 for sensing the rate of acceleration of themobile sensor 10 about its pitch axis.

Temperature and gain compensation is provided for the pitchaccelerometer 620 by block 623. An analog-to-digital (A/D) converter 622converts the analog pitch accelerometer 620 signal to a digital signalthat can be used by the microprocessor 606. The A/D converter 622 alsoconverts the analog pitch gyroscope 621 signal to a digital signal.

It should be noted that the terms roll, yaw, and pitch are usedthroughout this specification as a means of distinguishing each of thethree axes about which the unit can move, and is not intended to implythat the roll accelerometer 610 is capable of only measuring rotationabout an object's roll axis, and so on. Depending on how the mobilesensor 10 is mounted or held during a trip, the roll accelerometer 610may actually be measuring the magnitude of acceleration on the object'spitch or yaw axes. This is also true for the yaw accelerometer 600, thepitch accelerometer 620, the roll gyroscope 611, the yaw gyroscope 601,and the pitch gyroscope 621.

The power board 53 includes a charger connector 640 for interfacing toan external power source such as a wall charger. This charger connector640 is isolated from causing damage to the power board 53 by an overloadprotection circuit 641. The power board 53 includes a plurality ofvoltage regulators and references 642, 643, 644, and 648 for supplyingpower to the various circuit functions on the mobile sensor 10. Acharging and power management circuit 647 is provided to oversee thecharging of the rechargeable power source 52 and to selectively disablemobile sensor 10 functions in order to prolong battery life. A switchdebounce and overvoltage protection circuit 646 is provided to preventnoisy user input lines from causing inadvertent feature activations.Finally, a barometric pressure transducer 645 is provided to detectchanges in ambient barometric pressure, allowing the mobile sensor 10 tocalculate changes in altitude.

A decal and switch panel 11 and indicator lights 51 are provided forinterfacing with the operator. The indicator lights 51 include statusindicator lights 630, an indicator driver circuit 631, and a separatecharge status indicator light 632 that is tied directly to the chargingand power management circuit 647 on the power board 53 to indicate thecharge status of the rechargeable power source 52.

A wireless radio module 56B provides a mechanism for downloading thedata stored in the storage device 609 to an external system via awireless data connection. Alternate embodiments of the mobile sensor 10may also use a direct-wired connection such as RS-232 or a removablememory device 673 to transfer data.

The satellite receiver board 54 includes an antenna 670 to increasereception, a satellite receiver module 671, a backup voltage regulator672, a removable memory module 673 such as a Flash Multi-Media Card(MMC) or a Secure Digital (SD) card, and a voltage level translator 674that allows the features on the satellite receiver board 54 to interfaceto the microprocessor 606.

FIG. 7 shows an example of a representative graphical user interface(GUI) for a flight analysis application that executes on a separatedesktop or handheld computer. This flight analysis application processesthe data captured by the mobile sensor 10, performs any correctionaladjustments required to the data, creates a three-dimensionalrepresentation of the motion of the sensor corresponding to the data,and displays the recreated event on the computer monitor. The featuresdescribed herein are examples only and are not meant to limit thefunctionality in any manner. The main window 70 is a typical graphicaluser interface (GUI) window. A set of pull-down menus 71 provides a listof typical commands and command types. A synthetic vision window 72A isdedicated to displaying the recreated playback on a syntheticthree-dimensional environment, which may include actual satellite orhigh-altitude photos of the environment where the data was recorded. Asimulated gauge panel 72B provides a functioning set of simulatedaircraft gauges and instruments. A portion of the screen is dedicated tothe display of specific data parameters, including the parameter labels73A and text boxes 73B containing the numeric values associated withthese parameters. Another portion of the screen is dedicated toproviding alternate views of the playback to the operator, includingbutton controls featuring default “camera angles” 74A, button controlsused to toggle display items 74B on and off, and a tab control device74C for selecting between three-dimensional (3D) viewing of the data andtwo-dimensional (2D) viewing of the data. VCR-style controls 75 (such asforward, reverse, play, and pause) are provided to allow the operator tomove backward and forward through the playback at will, and a progressindicator bar 76B is provided to indicate the current position in theplayback, as well as to act as a slider control for moving to any pointin the playback. A vertical zoom slider bar 76A is provided to move the“camera” in to and out from the aircraft during the playback. Additionaldata displays 77 provide information to the user, such as currentplayback speed, a time readout for the current playback, and the numberof graphics frames per second being displayed.

FIG. 8 shows the same example graphical user interface (GUI) as shown inFIG. 7 with changes to represent how the flight analysis applicationmight appear when the data is displayed in two-dimensional mode, orgraph mode. Only the features that have changed from FIG. 7 have beennumbered in FIG. 8, and all other features should be consideredidentical to FIG. 7. Again, the features described herein are examplesonly and are not meant to limit the functionality in any manner.

A graph window 80 is displayed with a grid pattern 82 representing unitsof playback time and data value magnitude. Graphical plots 81 of severaldifferent flight parameters are plotted against the grid pattern 82,corresponding to actual data values seen during the recorded event.Parameter labels 83 are provided to show the actual numeric value at thecurrent point in the playback. Graph line controls 84 appear intwo-dimensional mode to allow the user to select which plot lines appearon the graph window 80. Graph item controls 85 appear to allow the userto toggle the display of certain graph items on or off

FIG. 9 shows the same example graphical user interface (GUI) as shown inFIG. 7 and FIG. 8 with changes to represent additional graphicalfeatures available during the three-dimensional (3D) playback. Thesynthetic vision window 72A again shows a playback of a recorded flighton a three-dimensional recreation of the environment in which the datawas recorded. A model of the aircraft 91 is displayed at a position andorientation corresponding to the position and orientation of the actualaircraft. A data ribbon 92 extends behind and in front of the aircraftshowing the recorded flight path. A checkerboard altitude wall 93provides a graphical representation of the altitude of the aircraft,where each square of the checkerboard pattern represents a pre-definednumber of feet of both horizontal and vertical distance.

FIG. 10 is a high-level flowchart showing the flow of control requiredon the mobile sensor 10, the desktop application running on the desktopcomputer 20 or the handheld computing device 30, and the centralizedserver 22 during a typical record and playback cycle. Processing startsin “Begin Operate Mobile Sensor” 1000, which represents the operatorturning the mobile sensor 10 on. A calibration procedure 1001 istypically required to initialize the mobile sensor 10 to a known state.The mobile sensor 10 must then acquire a signal lock on the GPSsatellite 1002 in order to begin recording satellite data. Oncesatellite lock 1002 is obtained, the mobile sensor 10 must wait for theuser to press the record button 1003 and 1004, after which it begins toacquire data 1005 via the on-board sensors. This data is stored locallyin the on-board memory 1006 until the operator presses the Record buttona second time to turn off the record function 1007. After the recordfunction is terminated 1007, the mobile sensor 10 waits until a datadownload is commanded 1008 and 1009, and then downloads the data to thedesktop system 1010 via a data transfer means 1023, which may include adirect-wired connection, a wireless connection, or data transfer bymeans of a removable memory device, thereby ending the “acquire data”operation of the mobile sensor 1011. The downloaded data is stored onthe desktop application in a trip file database 1022.

Processing for the desktop application begins in “Begin Operate DesktopApplication” 1012, representing the operator executing the desktopapplication. The desktop application loads the trip file 1013 from thetrip file database 1022 and begins post-processing the data 1014,depending on stored readings from multiple sensor functions integral tothe mobile sensor to create a highly accurate trip data file. Based onthe geographic coordinates stored in the data file 1015, the desktopapplication then downloads one or more satellite or high-altitude imagescorresponding to the data file 1016 from an external image/map databaseon a centralized server 1021 or over an internet connection 1024. Thedesktop application then creates a synthetic representation of theenvironment 1017, displays the created trip visualization on the monitor1018, and then responds to operator inputs via the playback controls andapplication commands 1019. The process terminates with “End OperateDesktop Application” 1020, which represents the operator terminating thedesktop session and exiting the software.

FIGS. 11, 12, and 13 provide definitions of the terms yaw, pitch, androll, respectively, and are not otherwise referenced in the text of thisspecification. These terms are used throughout the specification and itis important that they are fully understood in this context.

FIG. 11 provides a definition of the term yaw, and shows a top view of amoving body 1100 such as an aircraft. The yaw angle 1103 is the numberof degrees measured between the course 1102 of the moving body 1100 andthe heading 1101 of the moving body 1100. The course 1102 of an objectis defined to be the actual direction of movement of that object, andthe heading 1101 is defined to be the direction that the object isfacing. The yaw axis 1104 is the point about which the moving body 1100rotates when demonstrating a change in yaw.

FIG. 12 provides a definition of the term pitch, and shows a side viewof a moving body 1100 such as an aircraft. The pitch angle 1203 is thenumber of degrees measured between the “level” orientation of flight1202 for the moving body 1100 and current orientation 1201 of the movingbody 1100, as the moving body 1100 rotates about the pitch axis 1204.

FIG. 13 provides a definition of the term roll, and shows a front viewof a moving body 1100 such as an aircraft. The roll angle 1303 is thenumber of degrees measured between the “level” orientation of flight1302 for the moving body 1100 and current orientation 1301 of the movingbody 1100, as the moving body 1100 rotates about the roll axis 1304.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the preferred embodiment, the flight training and syntheticvisualization system is used primarily as a flight training aid,providing playback and analysis of flight data recorded by a mobilesensor (this embodiment is illustrated in FIG. 2). A user mounts themobile sensor 10 in or on an aircraft or other moving object (the movingobject could also be a person such as a skydiver). The mobile sensor 10is turned on, the Record button is pressed, and recording begins. Onceoperational, the mobile sensor 10 follows the algorithm described inFIG. 10 (Steps 1000 through 1011), acquiring flight data describing theposition and orientation of the mobile sensor 10 as it moves throughthree-dimensional space.

While it is recording, the mobile sensor 10 relies on a plurality ofon-board sensors to obtain flight data. In the preferred embodiment(FIG. 6), the mobile sensor 10 comprises:

-   -   a yaw accelerometer 600, a roll accelerometer 610, and a pitch        accelerometer 620 to record the magnitude of acceleration of        movement in three dimensions,    -   a yaw gyroscope 601, a roll gyroscope 611, and a yaw gyroscope        621 to record the rate of acceleration of movement in three        dimensions,    -   two magnetoresistive compasses 604A and 604B to record the        magnetic heading by measuring the Earth's magnetic field,    -   a barometric pressure transducer 645 to measure the ambient        barometric pressure,    -   a wireless radio module 56B to allow the mobile sensor 10 to        communicate bi-directionally and wirelessly with the computer 20        hosting the desktop application,    -   a satellite receiver board 54 to allow the mobile sensor 10 to        receive transmissions from the global positioning system,    -   removable memory 673 as an alternate means of transferring data        between the mobile sensor 10 and the computer 20 hosting the        desktop application,    -   permanent on-board memory 609 for storing the flight data as it        is recorded,    -   a rechargeable power source 52 to provide wireless power to the        mobile sensor 10, and

user feedback devices in the form of a plurality of buttons 11 and aplurality of indicator lights 51.

Using this preferred electronic architecture, the mobile sensor 10records all movement and changes in orientation and stores this data inthe on-board memory 609 for later transmission to the computer 20. Inthis embodiment, the mobile sensor 10 does very little processing of thedata. This data is simply stored and later transferred to the computer20 where the desktop application will perform post-processing of thedata before playback.

Alternate embodiments of the mobile sensor 10 can be created with asmaller number of on-board sensors. While this would lower the accuracyof the data obtained, this approach would produce data that would besufficient for many applications that do not require sophisticated orhighly accurate monitoring of movement (such as the tracking ofland-based vehicles) and would result in a lower cost sensor.

Additional alternate embodiments of the mobile sensor 10 could becreated by adding additional sensors or additional data inputs via theoptional radio to the preferred embodiment. In this manner informationsuch as engine performance characteristics, waypoints, etc., could beadded to the stored data set for later retrieval. These additionalinputs could be added based on the specific needs of any application.

Once the mobile sensor 10 has finished recording a flight or trip, theoperator can terminate the recording process. The mobile sensor 10 canthen be turned off or set up to record another flight. Data alreadyrecorded will be maintained indefinitely in the on-board memory 609 orin the optional removable memory 673, until such time as the data can bedownloaded to the computer 20 hosting the desktop application.

When all flights or trips have been recorded, the user can transfer thedata from the mobile sensor 10 to the computer 20 using either thewireless or hardwired communication link 21, or, if so equipped, bytaking the removable memory device 673 out of the mobile sensor 10 andbringing it by hand to the computer 20. In any event, the data istransferred to the computer 20 and stored in a trip database 1022.

Additional alternate embodiments of the mobile sensor 10 could also becreated by using combinations of different memory devices and datatransfer means. Versions of the mobile sensor 10 could contain permanenton-board flash memory 609, a removable memory device such as an MMC card673, or both. The mobile sensor 10 could also have no on-board memorymeans and simply transfer the data immediately to an external device,such as the desktop computer 20.

Upon request by the user, the desktop application running on thecomputer 20 will load the trip data file 1013 and begin post-processingthe data 1014. This post-processing consists of analyzing the valuesgathered by multiple, redundant sensors (as described in FIG. 6) andcomparing and combining the values to achieve a data accuracy that wouldnot be attainable by any single sensor alone. For example, if there is agap in the GPS data received by the mobile sensor 10 (perhaps when thesatellite data is unavailable for a period of time), the movementsrecorded by the accelerometers (600, 610, and 620) and gyroscopes (601,611, and 621) can be used to fill in the gaps. In addition, changes inbarometric pressure detected by the barometric pressure transducer 645can be used by the mobile sensor 10 to calculate changes in altitude,which can supplement or replace the altitude derived from GPS data andinertial measurement sensors.

By transferring this processing activity from the mobile sensor 10 tothe desktop computer 20, the system can take advantage of the processingpower inherent in a typical desktop computer and off-load the processingburden from the mobile sensor 10 thus reducing the cost and complexityof the mobile sensor 10.

Once the post-processing 1014 has been completed, the desktopapplication uses the geographic coordinates stored in the data file 1022to calculate the area of the Earth's surface for which a satellite oraerial image is required. It then interfaces to an image/map database1021 on a centralized server over an internet-style connection 1024 anddownloads a satellite or aerial photo (or series of photo tiles) thatcorresponds to the geographic location 1016 and creates a realistic,three-dimensional graphic visualization 1017 of the aircraft (or movingobject) and its immediate environment. The desktop application thenresponds to user inputs 1019 allowing the user to play back the tripvisualization as one would play a movie on a DVD player.

A typical embodiment of the user interface for the desktop applicationis shown in FIGS. 7, 8, and 9. A typical embodiment of the desktopapplication would provide an area on the screen for thethree-dimensional playback 72A as well as simulated flight instruments72B, an area of text boxes 73A and 73B showing dynamic readouts ofimportant flight parameters, operator controls 74A, 74B, and 74C toallow the operator to control the angle at which the playback is shown,and DVD-style playback controls 75. In addition, data sets recorded bymultiple mobile sensors, such as those used by a team of skydivers,could be superimposed on the same three-dimensional playback 72A toallow for performance comparisons. Airport-specific data, such asapproach plates and glideslope and localizer paths, can be superimposedon the flight playback to allow a pilot to see how they performed duringa landing. Graphical devices can be used to show the status of certainflight parameters. For instance, a three-dimensional graph of anairplane's altitude can be shown in the form of a checkerboard wall 93that is displayed between the ground and the model of the aircraft 91 inthe playback, where each square on the checkerboard represents a certainnumber of feet in altitude or horizontal distance. A secondary ghostimage of the aircraft model 91 could be displayed on thethree-dimensional playback 72A to show variance from an ideal flightpath such as the approach path of an airport. Visualizations of specialairspace types, such as restricted flight zones or aerobatic performanceboxes, could be superimposed on the three-dimensional playback 72A.Simulated weather patterns can be created to match actual weatherconditions that existed at the time of the flight.

The desktop application can also be used to display data on the flightin two-dimensional graph mode 80. In two-dimensional graph mode 80, plotlines of the flight parameters 81 and current value labels 83 aredisplayed on a graph-like grid pattern 82 to allow for the analysis ofthe flight.

In an alternate embodiment of the flight training and syntheticvisualization system (FIG. 3), the mobile sensor 10 is used to gatherflight data that is displayed in real-time (while the trip is ongoing)on a portable laptop or handheld computing device 30. In thisembodiment, the system would be used primarily as a visual flight aid toprovide additional flight data and analysis to a pilot while the flightis in progress.

The handheld device 30 would be co-located with the mobile sensor 10 andwould transfer data in real-time over a wireless data connection 31. Theapplication running on the handheld device 30 would be similar to theapplication running on the desktop computer 20, but in most cases wouldnot have a connection to a centralized database. A realistic graphicaldepiction of the flight in progress would be displayed on the handhelddevice 30, allowing the pilot to view their ongoing flight from anyangle and to display analytical information during the flight. Satelliteimages could be pre-loaded to the handheld device 30 by the user beforethe flight, or a grid or similar artificial background could be used forthe real-time playback.

1. A method of detecting, recording, coprocessing and simultaneously displaying aircraft flight data and corresponding terrain data, which method comprises the steps of: providing a self-contained mobile data recording unit (MDRU) on the aircraft; providing said MDRU with an MDRU microprocessor; gathering with said MDRU microprocessor flight data including navigation and flight information captured by said MDRU; providing said MDRU with a computer readable media and storing said navigation and flight information on said MDRU computer readable media; computing a 3-D recreation of a flight path of the aircraft based on said navigational and flight information; computing a digital terrain model for an area of the Earth's surface including at least a portion of the flight path; generating a 3-D display of said 3-D recreation including: said terrain model; a representation of the aircraft superimposed on the terrain model; and a data ribbon representing the flight path superimposed on the terrain model; computing altitude readings from said navigational and flight information at pre-defined intervals along the flight path; using said altitude readings and said navigational and flight information to compute a 3-D display comprising a vertical synthetic flight wall extending downwardly from said flight path data ribbon to a ground level on said terrain model; subdividing said flight wall graphically into a vertically-oriented checkerboard configuration comprising multiple rectangular segments separated by multiple, horizontally-spaced vertical striations each representing a pre-defined horizontal distance and multiple, vertically-stacked horizontal striations each representing a pre-defined vertical distance, said pre-defined vertical and horizontal distances corresponding to altitude and distance of travel along said flight path respectively; dynamically displaying in 3-D on said display device said flight wall including said vertical and horizontal striations below said flight path data ribbon; dynamically displaying in 3-D on said display device with said graphics software engine the progress along said flight path of the aircraft on top of said flight wall and over said terrain model; and dynamically displaying aircraft altitudes at respective rectangular segments along said flight path.
 2. The method of claim 1, which includes the additional steps of: providing an inertial measurement sensor (IMS) on the aircraft; sensing orientation of the aircraft and generating orientation signals representing its orientation with said IMS; inputting said orientation signals to said computer; and computing said flight path using said orientation signals.
 3. The method of claim 2 wherein said IMS includes yaw, pitch and roll sensors, which method includes the additional steps of: continuously sensing 3-D orientation of the aircraft with said IMS; generating orientation signals with said IMS corresponding to the yaw, pitch and roll of the aircraft; and outputting said orientation signals from said IMS as input to said microprocessor.
 4. The method of claim 1, which method includes the additional steps of: generating 3-D, GNSS-based aircraft position signals; inputting said aircraft position signals to said microprocessor; and computing a data ribbon representing said flight path using said aircraft position signals.
 5. The method of claim 1 wherein said 3-D display comprises a moving video representation of said aircraft progressively moving along said flight path data ribbon.
 6. The method of claim 1, which includes the additional steps of: interfacing said microprocessor with an image/map database including 3-D terrain images comprising satellite or aerial photos or photo tiles; computing with said microprocessor an area of the Earth's surface including at least a portion of the flight path; and downloading satellite or aerial photos or photo tiles for said calculated area of the Earth's surface from said image/map database to said microprocessor.
 7. The method of claim 1, which includes the additional steps of: providing said computer with a graphics software engine; gathering weather pattern information; creating with said graphics software engine a display comprising a graphical representation of said weather pattern information; and displaying with said display device said graphical representation of said weather pattern information superimposed on said 3-D recreation.
 8. The method of claim 1, which includes the additional steps of: gathering special airspace type information; creating with said graphics software engine a display comprising a graphical representation of said special airspace type; and displaying with said display device said graphical representation of said special airspace type superimposed on said 3-D recreation.
 9. The method of claim 1, which includes the additional steps of: collecting navigational and flight information in-flight with said computer; providing a secondary computer; connecting said secondary computer to said aircraft computer; and post-processing said navigational and flight information and displaying said 3-D recreation with said secondary computer after a flight is completed.
 10. The method of claim 9 wherein said secondary computer comprises a handheld device and said method includes the additional step of: processing in real time during said flight said navigational and flight information and displaying said 3-D recreation on said handheld device.
 11. The method of claim 1 wherein said aircraft representation comprises a primary aircraft model, which method includes the additional steps of: generating a secondary ghost image model of the aircraft; computing an ideal flight path corresponding to locations of said secondary ghost image aircraft model; superimposing on said 3-D display said secondary ghost image model in relation to said primary aircraft image model; and displaying with said display device discrepancies between said ideal flight path and the actual flight path by simultaneously and dynamically displaying the positions of said primary and secondary aircraft image models relative to each other and the flight wall.
 12. A method of sensing and computing aircraft flight data associated with an aircraft flight path, and simultaneously displaying an aircraft flight path, flight wall and terrain model for a flight, which method comprises the steps of: generating signals representative of a 3-D, GNSS-based flight path of said aircraft with said position detector; providing a computer including a microprocessor on the aircraft and connected to the IMS and the position detector; providing a digital terrain model of a portion of the Earth's surface including at least a portion of the flight path; computing a 3-D display of said flight path including: said terrain model; a primary aircraft image model superimposed on and simulating movement relative to the terrain model; and a data ribbon representing the flight path superimposed on the terrain model; computing with said altitude readings and said navigational and flight information a 3-D vertical flight wall extending downwardly from said flight path data ribbon to a ground level on said terrain model; said flight wall being subdivided graphically into a checkerboard configuration comprising multiple rectangular segments each representing a pre-defined horizontal and vertical distance corresponding to altitude and distance of travel along said flight path respectively; providing a display device and connecting said display device to said microprocessor; displaying with said display device said 3-D display with said primary aircraft image model on top of said flight wall and said aircraft altitudes over said terrain model; generating a secondary ghost image model of the aircraft; computing an ideal flight path corresponding to locations of said secondary ghost image aircraft model; superimposing on said 3-D display said secondary ghost image model in relation to said primary aircraft image model; displaying with said display device discrepancies between said ideal flight path and the actual flight path by simultaneously and dynamically displaying the positions of said primary and secondary aircraft image models relative to each other and the flight wall; collecting navigational and flight information in-flight with said computer; providing a secondary computer; connecting said secondary computer to said aircraft computer; and post-processing said navigational and flight information and displaying said 3-D recreation with said secondary computer after a flight is completed.
 13. A system for simulating an aircraft flight, which system comprises: an inertial measurement sensor (IMS) installed on the aircraft and adapted for sensing orientation of the aircraft and generating orientation signals; a global navigation satellite system (GNSS) position detector installed on the aircraft and adapted for generating signals representative of a 3-D, GNSS-based flight path of said aircraft; a computer including a microprocessor installed on the aircraft and connected to the IMS and the position detector; said microprocessor being connected to and receiving input signals comprising navigational and flight information from said IMS and said GNSS position detector; said computer being adapted for receiving a digital terrain model of a portion of the Earth's surface including at least a portion of the flight path; said computer being adapted for computing a 3-D display of said flight path including: said terrain model; a model of the aircraft superimposed on and simulating movement relative to the terrain model; and a data ribbon representing the flight path superimposed on the terrain model; said computer being adapted for computing altitude readings from said navigational and flight information at predetermined intervals along the flight path; said computer being adapted for computing with said altitude readings and said navigational and flight information a 3-D vertical flight wall extending downwardly from said flight path data ribbon to a ground level on said terrain model; said flight wall being subdivided graphically into a checkerboard configuration comprising multiple rectangular segments each representing a pre-defined horizontal and vertical distance corresponding to altitude and distance of travel along said flight path respectively; and a display device connected to said microprocessor and adapted for displaying said 3-D display with said aircraft model on top of said flight wall and said aircraft altitudes over said terrain model.
 14. The aircraft flight simulation system according to claim 13, which includes: a graphics software engine installed on said computer and adapted for generating said 3-D display including a moving video representation of the aircraft progressively moving along the flight path data ribbon.
 15. The aircraft flight simulation system according to claim 13, which includes: said computer comprising a primary computer; and a secondary computer adapted for connection to said primary computer and post-processing sent navigational and flight information and displaying said 3-D recreation after a flight is completed.
 16. The aircraft flight simulation system according to claim 13 wherein said secondary computer comprises a handheld device adapted for processing in real-time during said flight said navigational and flight information and displaying said 3-D recreation on said handheld device. 